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ABSTRACT 


During  the  past  decade,  the  phrase  "management  information 
systems"  or  the  acronym  "MIS"  has  become  increasingly  popular 
for  discussions  of  information  processing  support  for  management 
activities.  Primary  developments  in  the  field  of  MIS  have  been 
paced  by  advances  in  computer  system  technology.  The  rationale 
and  economic  utilization  of  this  technology  within  management 
organizations  has  posed  some  fundamental  issues  for  system  design, 
particularly  among  professionals  who  view  "design"  as  a  normative 
science.  This  essay  reviews  these  issues  as  they  relate  to  the  stage 
in  design  called  "evaluation".  Specifically:  What  does  the  literature 
propose  on  MIS  evaluation  and  what  is  being  done  in  practice?  What 

is  the  state-of-the-art  today?  What  problems  loom  ahead? 

* 

1.  Introduction  and  Overview 

In  the  spring  of  196E  Professor  Herbert  A.  Simon  delivered  the 
Karl  Taylor  Compton  lectures  at  the  Massachusetts  Institute  of 
Technology  on  "the  sciences  of  the  artificial"  --  cf.  Simon 
(1969).  At  the  risk  of  oversimplification,  Simon's  thesis  in 
these  lectures  was  that  certain  phenomena  are  artificial  in  a 
very  specific  sense:  they  are  as  they  are  only  because  a  system 
has  been  molded,  by  goals  or  purposes,  to  the  environment  in 
which  it  lives.  By  way  of  contrast,  the  scientific  disciplines 
have  traditionally  concerned  natural  phenomena:  how  they  are  and 
how  they  work.  Today,  one  also  observes  studies  of  artificial 
phenomena:  how  to  make  artifacts  that  have  desired  properties 
and  how  to  design. 

What  is  the  process  called  "design"?  Is  it  art  or  science? 

In  simple  terms:  design  is  decisionmaking.  It  Involves  find* 
lng  alternatives  to  change  existing  situations  into  preferred 
situations.  For  the  engineer's  analogy  of  the  black  box,  it 
concerns  re-configuring  the  interval  environment  of  the  box. 
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so  that  given  inputs  yield  desired  outputs.  The  problem  focus 
in  design  is  the  interface  between  the  natural  laws  within  a 
system  and  the  natural  laws  without  it.  A  designer  is  concerned 
with  attaining  goals  by  adapting  characteristics  of  the  former 
to  die  latter,  and  the  process  by  which  the  adaptation  of  means 
to  environments  is  brought  about.  If  this  process  is  unique  to 
an  individual  and  cannot  be  promulgated,  the  design  is  art.  If 
through  research  and  understanding,  however,  we  establish  norma¬ 
tive  principles  to  guide  the  process  and  can  teach  these  rudi¬ 
ments  to  practitioners,  then  we  may  speak  of  a  "science  of 
design."  (To  quote  a  French  scholar:  "Art  is  I.  Science  is 
we.")  Today,  the  discipline  of  design  has  emerged  as  an 
"artificial  science"  in  common  with  such  professions  as  engineer 
ing,  architecture,  medicine,  law  and  management. 

During  the  past  decade,  the  phrase  "management  information 
systems"  or  the  acronym  "MIS"  has  become  increasingly  popular 
for  discussions  of  information  processing  support  for  management 
activities.  The  phrase  in  one  sense  is  more  descriptive  of  the 
end-user's  goals  for  the  information  system  (computer-based  and 
otherwise)  than  the  earlier  terminology  of  "business  data 
processing".  For  convenience  here,  I  forgo  a  discussion  of  the 
etymology  and  semantics  of  MIS,  and  will  assume  the  ph.rase  is 
synonymous  with  the  several  near  equivalents  appearing  in  the 
literature. 

Much  has  been  written  on  the  subject  of  MIS,  paritcularly  on 
the  process  of  system  design,  development  and  Implementation. 
Indeed,  most  introductory  textbooks  provide  flowcharts  in  an 
early  chapter  of  each  stage  in  systems  development  --  often  in¬ 
cluding  "cookbook"  recipes  and  checklists  of  "do's  and  dori'ts", 
occasionally  documented  by  the  author's  personal  experiences. 
Since  these  flowcharts  are  coentfnplace  and  reasonably  consistent 
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there  is  no  need  to  reproduce  them.  Rather  in  the  space  of  this 
essay,  I  wish  to  consider  one  aspect  of  the  design  and  develop¬ 
ment  process,  namely:  the  milestone  called  evaluation.  In 
particular,  given  a  conceptual  model  of  the  design  process  that 
involves  goal-directed  search:  What  does  the  literature  propose 
on  MIS  evaluation?  What  is  being  done  in  practice?  What  is  the 
state-of-the-art  today? 

2.  The  Problem:  Information  Economics 

The  issue  of  evaluation  is  conceptually  straightforward.  That 
is ,  one  seeks  to  observe  and  measure  output  (results)  within  the 
framework  of  some  "model"  which  describes  the  system  environment 
(and  interface)  in  terms  of  available  resources,  their  capacities, 
given  inputs,  and  criteria  of  performance  on  stated  goals.  As  a 
technical  matter  one  would  optimize  the  performance  criteria,  or 
at  least  for  any  pairwise  comparison  of  alternatives  prefer  the 
one  which  was  more  efficient  under  the  stipulated  measure  of 
performance.  Such  a  decision  paradigm  may  be  relatively  easy, 
say,  for  the  engineering  consideration  of  a  piece  of  capital 
equipment,  e.g.,  most  accountants  would  employ  a  net-present- 
value  criterion  for  the  discounted  "cash  flow"  of  dollars 
associated  with  the  investment.  The  decision  becomes  more  diffi¬ 
cult,  however,  when  the  investment  involves  a  configuration  of 
general  purpose  resources  and  the  end-use  alternatives  of  the 
resources  must  be  included  within  the  evaluation. 

Information  theory  in  communications  engineering  provides  a 
formal  framework  for  measuring  the  quantity  of  information  and 
communications  efficiency,  cf.  Hartley  (1928)  or  Shannon  and 
Heaver  (1948).  In  simple  terms  information  theory  is  concerned 
with  the  activities  of  a  "source"  who  encodes  a  massage,  a 
"communications  channel"  which  transmits  the  coded  mossage,  and 
a  "receiver"  who  decodes  tho  message  upon  receipt.  As  such  the 
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theory  says  nothing  about  the  activities  of  either  the  "source" 
or  the  "receiver"  outside  the  stated  symbol  processing  system  and, 
more  specifically,  it  tells  nothing  of  the  value  of  the  message 
communicated  to  the  end  user.  (For  example,  was  it  "Mary  had  a 
little  lamb"  or  "We  are  at  war"?) 

Decision  theory  in  economics  and  statistics  provides  a  formal 
framework  for  measuring  the  value  of  information  to  a  decision 
maker,  cf.  Wald  (1950)  or  Raiffa  (1968).  In  its  simplest  form, 
the  decision  theory  model  postulates  an  individual  who  must 
select  a  terminal  action  on  the  basis  of  either  current  informa¬ 
tion  or  additional  information  he  can  purchase,  the  outcome 
payoff  being  a  function  of  the  action  selected  and  an  uncertain 
event  outside  his  control.  This  single-person  model  has  been 
generalized  under  certain  restrictive  assumptions  on  behavior  to 
multi-person  organizations  through  a  framework  called  "the  theory 
of  teams,"  cf.  Marschak  (1955)  and  Radner  (1962).  More  recently, 
Marschak  has  attempted  to  link  the  team  decision  model  with  the 
communications  model  of  information  theory,  in  effect,  by  pre¬ 
ceding  the  "source"  with  an  "observer"  and  following  the  "receiver 
with  a  "decision-maker",  cf.  Marschak  (1968)  and  Chapter  3  in 
Kriebel,  et  al  (1970).  Other  variations  of  the  classical  decision 

theory  model  are  also  beginning  to  appear  in  the  literature  of 

1/ 

contemporary  economics.-* 

The  important  aspect  of  this  literature,  irrespective  of  model¬ 
ing  details,  is  that  decision  theory  provides  the  relevant 
theorem  on  Information  economics  for  system  evaluation.  Speci¬ 
fically,  the  expected  value  of  information  to  a  manager  (i.e., 

~^For  example,  see  the  Program  and  Abstracts  of  Papers:  Second 
World  Congress  of  the  Econometric  Society  (8-14  September  1973; 
Cambridge,  England),  particularly  pages  47,  60,  95,  104,  120, 
166,  187,  and  215. 
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decision  maker)  is  computed  from  the  expected  net  gain  (benefits 
less  cost)  of  his  decision  rule  (strategy)  including  the  informa¬ 
tion  less  the  rule  excluding  that  inf  rmation,  where  mathematical 
expectations  are  calculated  on  a  rational  basis  according  to  a 
priori  probabilities.  That  is,  information  has  value  if  its 
expected  net  gain  is  positive  when  computed  according  to  the 
algorithm.  To  implement  the  theorem  management  must  structure 
the  decision  problem  such  that  the  decision  mechanism  or  rule 
for  choosing  terminal  actions  is  made  explicit,  the  -space  of 
outcomes  is  specified  and  ranked  in  order  of  preference  (payoff), 
and  information  opportunities  on  uncertainties  are  encoded  in 
terms  of  probability  distributions.  Do  these  requirements  ef¬ 
fectively  preclude  application  of  the  procedure  in  any  complex 
real  world  environment?  * 

In  general,  I  think  the  answer  to  this  question  is:  no.  Cer¬ 
tainly,  the  broad  principle  of  positive  expected  value  is  rational 
and  consistent  with  our  ex  ante  intuition.  Furthermore,  for  many 
non-trivlal  situations  —  if  not  at  the  "total  MIS"  level,  per¬ 
haps  —  the  computations  and  analysis  can  be  performed  exactly. 
More  broadly,  however,  it  is  important  to  recognise  that  the 
comparative  gains  of  technology  in  Information  processing  have 
been  realised  only  by  formal  information  systems  in  organisations. 
That  is,  the  orders  of  magnitude  increase  in  data  processing 
efficiencies  have  been  realised  over  the  pest  decade  through 
capital-intensive  technology  applied  to  formalised  systems.  In  - 
contrast,  informal  information  systems  which  tend  to  be  labor- 
intensive  have  not  been  affected  by  the  economies  and  leverage 
of  computers.  For  the  foreseeable  future,  the  demand  for  infor¬ 
mation  processing  capacity  by  managers  and  organisations  sppears 
ever  increasing.  To  expend  the  supply  and  ability  to  service 
these  dtnunds  will  require  Increasing  reliance  on  formal  informa¬ 
tion  systems  snd  capital-based  technology.  Thus,  the  management 
information  systems  that  we  seek  to  design  and  evaluate  will 
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require  specificity  and  explicit  statements  of  decision  mechanisms, 
uncertainty  encoding,  and  the  like,  to  the  degree  that  they  be¬ 
come  formalized.  To  this  degree,  1  .think  the  decision  theory 
construct  for  information  value  serves  as  a  useful  criteria'  in 
design. 

Maybe,  1  have  overstated  the  case  for  theory  in  making  the  point 
for  economic  relevance.  Certainly,  the  task  of  incorporating  a 
representative  characterization  of  the  technology  into  a  formal 
model  is  a  major  hurdle  in  evaluation,  and  many  non-economic 
factors  will  influence  the  final  design.  For  example,  Figure  1 
might  be  considered  as  onte  macro-flowchart  which  diagrams  the 
"bare  bones"  structure  of  an  MIS.  Without  dwelling  on  the  details 
of  the  illustration,  it  is  apparent  that  people  are  prominently 
involved  as  symbol  processor's  in  many  stages  of  the  cycle,  and 
behavioral  considerations  will  often  dominate  given  situations. 
Figure  1  also  highlights  the  sub-system  role  of  data  processing 
technology  within  the  broader  designation  of  a  formal  MIS.  At 
this  juncture  it  may  be  constructive  to  briefly  consider  the 
problem  of  evaluating  the  technological  component  of  an  MIS 
before  returning  to  the  broader  issue. 

3.  The  Evaluation  of  Information  Processing  Technology 

Progress  in  building  information  processing  hardware  has  far 
outdistanced  progress  in  evaluating  the  performance  of  this 
technology  in  systems.  As  Sharpe  (1969)  recently  reamrked  In 
his  excellent  book,  it  Is  one  thing  to  write  out  a  series  of 
equations  which  purport  to  functionally  describe  a  computer 
configuration,  Its  another  matter  to  obtain  the  true  parameters 
and  functions  which  pertain  to  an  actual  configuration. 

For  example,  Henderson  (1960)  proposed  an  Integer  linear  pro¬ 
gramming  formulation  of  the  problem  of  data  processing  design 


Figure  1 
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for  an  organization.  In  Henderson's  model  data  requirements,  report 

content  and  interrelations,  data  flows,  processing  capacities  and 

technological  constraints,  etc.  were  to  be  specified  in  the  form 

of  constraints  and  the  linear  criterion  function  of  net  benefits 

was  to  be  maximized.  Perhaps  not  too  surprisingly,  Henderson's 

model  was  not  empirically-based  and  included  the  additional  caveat 

that  as  presented  the  model  was  too  large  for  existing  (1960) 

computers  and  known  linear-integer  programming  algorithms  —  the 

•  1/ 

latter  qualification  is  still  true  today.-*  Similar  linear  pro¬ 
gramming  models  for  computer  selection  have  also  appeared  in  the 
literature,  e.g.,  Schneidewind  (1966),  where  the  criterion  is 
cost  minimization  subject  to  given  performance  constraints.  To. 
date,  the  prospects  for  a  completely  objective  model,  such  as  LP, 
as  a  solution  procedure  do  not  appear  particularly  promising. 

Beyond  the  computational  problem,  the  biggest  obstacle  appears 
to- be  the  likelihood  that  the  user  (designer)  can  capture  all  of 
the  interrelations  and  empirical  measurements  needed  for  a  "state¬ 
ment"  of  performance  requirements.  For  example,  consider  the 
difficulty  of  specifying  the  requisite  detail  for  the  factors 
listed  in  Table  1.  A  survey  of  other  formalized  models  for  design 
and  evaluation  is  provided  in  Kriebel  (1967)  --  see  also  Sharpe 
(1969),  Chapters  8-11. 

At  the  opposite  extreme  of  the  completely  "objective"  mathematical 
model  of  the  total  system  is  the  subjective  approach,  perhaps 
exemplified  by  "competitive  bidding".  As  commonly  practiced 
today,  the  user  effectively .delegates  the  computer  configuration 
design  to  the  manufacturer  by  first  preparing  a  "statement"  of 
system  requirements  and  then  soliciting  competitive  bids  from 
vendors  on  various  system  alternatives.  In  this  regard  the  user 
may  attempt  to  employ  one  of  several  informal  decision  models  for 
the  cost/value  evaluation  of  proposals.  For  example,  (1)  the  user 
specifics  minimum  performance  and  selects  the  minimum  cost  system 


1/  Mote  the  reference  here  is  not  a  criticism  of  Henderson's  model, 
per  sc;  to  the  contrary  his  research  was  an  oarly  and  imaginative 
delination  of  the  design  problem. 
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Tablc  1 

DIMENSIONS  OF  SYSTEM  REQUIREMENTS  STATEMENT 
AND  CONSTRAINTS 


WORKLOAD:  Applications 

Processing  Functions 

Inputs,  Compilation,  Computation,  Control  Data 
Acquisition,  File  Maintenance,  Media  Conversion, 
Sorting,  Storage  and  Retrieval,  Outputs,  Pro¬ 
cessing  Frequencies,  Security /Privacy,  Relative 
Priorities 

Development/Implementation  Schedules 
Working  Programs 

Production  Operations  Constraints 

Data  Volumes,  Thruput, Turn- around  Times, 
-Response  Times,  User  Priorities,  Physical 
Space 

Etc.  * 

HARDWARE:  Units 

Channels,  Control  Units,  Direct-access  Devices, 
Storage,  Tape/Disk/Drum  Drives ,  Processors, 
Terminals,  Unit  Record  Devices 
Interfaces 

External  Communication  Networks,  Man/Machine 
Interface,  Other  Computers,  Recording  Media 

SOFTWARE:  Source  Language,  Object  Language 

System  Functions,  Supervisory  Programs,  Data 
Management,  File  Design,  Task  Control 

SYSTEM  ORGANIZATION: 

Conventional  batch  processing.  Multiprocessing  (Direct 
coupled  Many  Processors),  Multiprogramming  (Multi¬ 
tasking,  Telecommunications,  Multiple-access) 
Compatibility 

DATA  REPRESENTATION 
PERSONNEL 

OPERATIONS  ORGANIZATION  AND  ADMINISTRATION 

EDUCATION  AND  TRAINING 

POLICY  AND  PROCEDURES  . 

MANAGEMENT  INTERFACE 
ETC. 
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proposed;  (2)  the  user  specifics  maximum  cost  and  selects  the 
maximum  performance  system  proposed;  or  (3)  the  user  specifics 
minimum  performance  and  maximum  cost,  and  selects  the  system 
having  the  maximum  value/cost  ratio,  employing  tie-breaking  rules 
if  necessary.  A  completely  subjective  procedure  for  the  "com¬ 
petitive  bid"  approach  suffers  from  all  the  obvious  weaknesses 
of  human  judgment  and  error,  conflict  of  interest  with  other 
goals,  bias,  inefficiencies,  and,  so  on.  The  "statement"  of  re¬ 
quirements  in  this  case,  if  not  as  rigorous  as  implied  by  the 
total  system  model,  must  nevertheless  attend  to  many  of  the  same 
issues  listed  in  Table  1.  Furthermore,  the  imposition  of  any 
single  rigid  measure  of  performance  and/or  requirement  for  per¬ 
formance  as  a  short-cut  to  the  complete  specification  is  almost 
assured  to  lead  to  a  sub-optimal  (and  perhaps  grossly  inferior) 
result.  For  example,  consider  the  relative  uselessness  of  a 
single  performance  index  for  a  computer,  say  "cycle  time",  as  a 
basis  for  system  evaluation.  One  concludes  from  experience  that 
some  middle  ground  between  Idle  complete  model  and  a  subjective 
procedure  is  more  likely  to  lead  to  preferred  and  realistic 
results. 

Pursuing  the  middle  ground  to  technology  evaluation  the  most 
common  objective  methods  employed  to  guide  a  subjective  decision 
process  include:  figures  of  merit,  scoring  systems,  or  instruc¬ 
tion  mixes;  kernel  timing  estimates;  (so-called)  benchmark  pro¬ 
grams;  and  simulations.  The  first  method  may  range  from  a  fairly 
simple  expression  (such  as:  maximise  the  log  (memory  sise/cycle 
time})  for  performance  of  alternative  hardware,  to  more  lengthy 
formulas  which  subjectively  weight  non-hardware  characteristics 
(such  as,  adaptability,  supplier  support,  and  so  on)  —  cf., 

Solomon  (1966),  Bromley  (1965),  Arbuckle  .(1966)  and  Knight  (1963). 
Although  relatively  simple  to  use  the  difficulty  with  those  methods 
lies  in  their  highly  subjective  specification  and  myopic  view  of 
the  larger  problem.  For  example,  through  such  a  formula,  one  can 


obtain  the  result  that  doubling  the  capacity  of  high  speed  core 
will  double  the  thruput  of  a  system  —  which  is  absurd. 

l 

Kernel  timing  estimates  are  often  employed  for  cost  comparison 
among  computers  to  perform  a  given  task.  Calingaert  (1967) 
defines  a  program  kernel  as  "the  central  processor  coding  required 
to  execute  a  task  of  the  order  of  magnitude  of  calculating  a 
social  security  tax,  inverting  a  matrix,  or  evaluating  a  poly¬ 
nomial".  In  comparing  estimates  one  assumes  equal  efficiency 
and  sophistication  of  the  coding  in  assembly  language  for  the 
various  machine  alternatives,  cf.  IBM  (1965).  Benchmark  programs 
differ  from  kernels  in  that  they  typically  represent  programs 
(or  problems)  which  exist  on  a  current  system  and  for  which 
execution  times  are  desired  on  a  proposed  system.  Alternatively, 
the  Auerbach  Corporati«..V  considers  six  benchmark  problems  for 
evaluation  purposes:  updating  sequential  files,  updating  files 
on  random-access  storage,  sorting,  matrix  inversion,  evaluation 
of  complex  equations,  and  statistical  computations,  cf.  Hillegass 
(1966).  The  final  extension  of  replicating  "representative"  data 
processing  workloads  or  requirements  is  to  employ  simulation. 

In  addition  to  in-house  or  proprietary  simulators  several  commer¬ 
cially  packaged  simulations  have  appeared  on  the  market.  A 
summary  of  the  characteristics  of  three  such  packages  is  shown  in 
Table  2;  see  Bairstow  (1970).  While  simulation  packages  repre¬ 
sent  advance  stage  models  they  still  suffer  from  the  subjectivity 
of  requirements  specifications  by  the  user  (customer),  they  may 
present  biased  results,  and,  as  indicated  by  Table  2,  the  commer¬ 
cial  packages  are  relatively  expensive. 

Evaluating  hardware  technology  --  the  central  processor,  peripherals, 
and  configuration  alternatives  —  is  complex,  but  tho  inclusion 
of  software  performance  considerably  adds  to  the  scope  of  the 
problem,  cf.  Calingaert  (1967).  Even  though  the  "unbundling 


-12- 


-  Table  2 

REPRESENT  ATI  VE  COMMERCIAL  PACKAGES  OP  SIMULATORS 
FOR  COMPUTER  SYSTEM  EVALUATION 


.  TKOGKAM: 

CASE 

SAM 

SCERT 

NAME: 

Computer-Aided  System 
Evaluation 

System  Analysis 

Machine 

System  and  Computer 

Evaluation  and  Review 
Technique 

COMPANY: 

Computer  Learning  and 
Systems  Corp, 

3530  Wisconsin  Ave, 

Chevy  Chase,  Md.  20015 

Applied  Data  Research 
Route  206  Center 
Princeton,  N.J.  08540 

Conrris 

2  kcscareh  Court 

Rockville,  Md.  20850 

COSTS: 

Purchase 

Lease  (Monthly) 
Single  study 
Analyst  tupporc 

$50,0<)0 

$1,000  (1  yr.nln, )' 

$4,000  and  up 
$250/day 

Not  fot  tale 
$3,500  (3  no, min.) 

$5,000  and  up 
$250/day  (after  10  days) 

Not  for  sale  j 
$1,000-3,000  (1  yr.nin.)  ! 
$5,000  and  up  { 
$250/«lay 

$1,000  for  10  nan-days 
(Includes  10  nan-days  of 
analyst  support)  • 

5  man-days  free 

$500  for  10  man-days 

SYSTEM: 

Aval  Uhle  on 
Cere  needed 
Written  In 

360/50, GE  600,  CDC  6000, 
200K 

Fortran 

360/50 

225K 

Fortran 

360/40  (and  up),  L'nivac  1108 
1I0K 

Assembler 

DATE  AVAILABLE: 

1969 

To  be  released 

1964 

decision"  by  the  key  manufacturers  last  year  to  price  hardware 
and  software  separately  appeared  at  first  glance  to  simplify  ; 
some  of  these  issues,  the  software  marketplace  is  still  a  morass 
of  confusion.  For  example,  during  the  past  few  years  a  major  con¬ 
sideration  in  software  design  and  development  for  MIS  has  been  the 
appearance  of  "data  (file)  management  systems"  or  "generalized  data 
base  management  systems";  e.g.,  Byrnes  and  Steig  (1969),  and  Fry 
and  Gosden  (1969).  In  broad  terms,  data  management  software  is 
Intended  to  de-couple  the  user  from  the  technical  details  of  the 
hardware  in  Interfacing  with  a  data  bank;  as  such,  it  may  be  char¬ 
acterised  as  an  extension  of  programming  language  capability  for 
data  manipulation.  Today,  however,  there  are  more  than  fifty  such 
software  systems  conmercially  available  (cf.,  CODASY^,  May  1969)  to 
the  bewildered  customer,  each  offering  a  variety  of  features  and  ap¬ 
proaches  to  data  management,  with  relatively  little  hope  (cf.  CODASYt, 
October  1969)  for  cosnonallty  or  standards  in  the  foreseeable  future. 
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Figure  2 

.  TREND  COMPARISON  OF  COMPUTER  SYSTEM 
COST  DISTRIBUTION  IN  TYPICAL  COMPANY 


Beyond  specific  software  technology,  trends  over  the  past  decade 
strongly  indicate  that  the  largest  cost  element  of  systems  is 
"people."  Figure  2  summarizes  the  proportionate  cost  distribu¬ 
tion  since  1957  and  projects  the  likely  ratio  for  1972.  While 
relative  costs  are  fairly  easy  to  extrapolate  it  is  clear  from 
industrial  experience  that  cost  estimates  for  application  pro¬ 
jects,  particularly  personnel  cost  estimates,  are  usually  poor 
and  biased  downwards,  (e.g.,  sqe  Laska  (1970)  for  some  unfortunate 
case  histories).  Thus,  one  concludes  that  the  state-of-the-art 
in  evaluating  system  technology  today  is  still  very  subjective 
and  to  some  degree  the  dark  art  of  "magic".  As  a  science 
there  is  much  need  for  empirical  research  —  for  those  interested 
in  more  extensive  bibliographies  see  Buchholz  (1969)  and  the 
Metametrics  Corporation  (1969),  as  well  as  earlier  citations.- 
Rather  than  despair  the  issue  entirely,  it  might  be  argued  with 
caution  that  technological  evaluation  is  not  the -crux  of  the 
problem  in  MIS.  Indeed  the  management  potential  of  a  particular 
system  application  may  provide  benefits  of  a  magnitude  that 
overwhelm  the  need  for  a. detailed  appraisal  of  technological  costs. 
Why  sub-optimize  the  technological  design  when  the  primary  con¬ 
siderations  in  MIS  are  management  opportunities? 

4.  MIS  Evaluation:  The  State-of-the  Art 

Parhaps  the  most  Important  lesson  of  the  past  decade  in  the  field 
of  MIS  has  been  the  maxim  that  the  relative  success  of  e  system 
application  is  a  direct  function  of  the  participation  by  manage¬ 
ment  in  its  design  and  development.  McKinney  (1968)  recently 
summarised  this  fact  in  an  International  survey  of  industrial 
corporations,  by  noting  three  dimensions  of  management  eystms 
performance:  technical  feasibility,  economic  feasibility  and 
operational  feasibility*  The  third  dimension,  conspicuously 
absont  in  system  failures,  roughly  translates  into  the  requirement 
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that  the  developed  system  be  understood  and  used  by  managers. 

Given  that  a  system  (application)  has  successfully  met  the  tests 
of  technical  and  economic  feasibility,  "will  managers  adapt  to 
the  system,  or  will  they  resist  or  ignore' it?"  By  analogy  with 
Professor  Churchman's  (1968)  perspective  on  "systems  analysis," 
the  principals  involved  in  design  and  development,  technicians 
and  managers,  must  be  sensitive  to  each  other's  value  system  — 
they  should  try  to  see  the  world  through  one  another's  eyes. 

Having  acquired  this  "sensitivity,"  they  should  put. their  rela¬ 
tive  expertise  in  perspective. 

Within  a  large  organization  computer  and  data  processing  staff 
can  be  expected  to  possess  a  reasonable  degree  of  technical  com¬ 
petence,  given  educational  and  experience  credentials.  Staff 
technicians,  however,  rarely  possess  the  broad  guage  focus  of 
line  management  in  understanding  the  economic  and  environmental 
factors  critical  for  the  particular  organization's  successful 
enterprise. 

One  approach  to  the  goals  and  criterion  problem  is  to  seek  a 
general  strategy  statement  from  management  for  Information  systems 
design  and  development,  e.g.,  Kriebel  (1968).  Another  technique, 
that  has  received  acceptance  in  a  variety  of  forms,  is  illustrated 
by  the  so-called  "Study  Organisation  Plan  (SOP)"  of  IBM  (1963). 

In  brief  SOP  approaches  a  systems  study  in  three  phases: 

(1)  understanding  the  present  business,  (2)  determining  system 
requirements,  and  (3)  designing  the  now  system.  The  first  phase 
of  the  study  seeks  a  detailed  description  of  the  "present  busi¬ 
ness"  through  an  economic  analysis  which  includes:  history  and 
framework,  industry  background,  firm  goals  and  objectives,  firm 
policies  and  practices,  government  regulations,  products  and 
markets,  materials  and  suppliers,  and  resources  (facilities, 
personnel,  inventory,  and  financial).  This  description  is  than 
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summarized  into  a  list  of  "activities"' which  arc  costed.  Phase 
2  of  SOP  analyzes  the  goals  and  objectives,  economics,  and  pro¬ 
cedure  of  execution  for  each  activity  interns  of  input-output 
requirements  in  operational  detail.  The  final  phase  of  SOP 
develops  specific  recommendations  and  plans  for  introducing  a 
new  business  system.  IBM  gives  five  criteria  for  choosing 
activities  as  automatic  data  processing  candidates,  viz.: 

(1)  Dominant  performance  criteria  (e.g.,  response  time  to  cus¬ 
tomer  inquiry).  (2)  High  af fee  table  dollar  savings.  (3)  Large 
data  processing  size  (e.g.,  volumes?.  (4)  Inefficiencies. 

(5)  Management  preference.  The  "technique"  includes  some  well- 
designed  forms  to  assist  conduct  of  the  study  at  each  phase,  and 
although  the  ideas  are  not  "new"  the  logic  is  well-founded  and 
useful  in  practice. 

t 

> 

In  the  absence  of  a  direct  policy  statement  by  management  or  an 
extensive  economic  analysis  of  "the  business",  perhaps  the  most 
important  index  of  the  relative  worth  of  a  system  application  is 
the  criticality  of  the  activity  to  the  organisation.  Sometimes 
the  index  can  be  measured  directly,  e.g.,  cost  savings  or  pro¬ 
fitability;  however,  often  it  in  a  function  of  surrogate  measures. 
The  role  of  surrogates  as  proximate  criteria  of  performance  may 
often  be  the  only  alternative  available  to  the  designer,  particu¬ 
larly,  if  the  system  application  is  not  directly  tied  to  a  manage¬ 
ment  activity  --  e.g.,  file  maintenance.  Surrogate  criteria  can 
also  serve  in  performing  a  "dominance  analysis"  of  the  attributes 
or  properties  of  design  alternatives,  in  lieu  of  a  uni-dimensional 
objective  function.  For  example,  the  development  of  proximate 
criteria  can  serve  to  establish  bounds  and  constraints  on  the 
search  for  design  alternatives  in  much  the  same  manner  as  a  con- 
stsaer  product  testing  organisation's  report  of  "best  buys"  for 
household  items.  In  this  regard  the  decision  theory  criterion 
may  provide  a  useful  guideline  for  directing  search. 
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It  is  clear  today  that  the  magnitude  of  the  problem  of  MIS  eval¬ 
uation  is  going  to  grow  during  the  next  decade  and  beyond.  In 
the  late  1950's  and  early  1960's  clerical  displacement,  cost 
reduction  and  control  of  administrative  expense  were  the  primary 
criteria  applied  to  management  data  processing  system  proposals. 
The  direction  and  scope  of  MIS  has  long  since  left  the  domain  of 
administrative  and  accounting  applications.  Last  year  Diebold 
(1969)  projected  a  time  frame  for  applications  and  evaluation 
criteria  over  the  next  15  years.  These  forecasts  are  sumnarized 
in  Table  3.  Referring  to  the  criteria  column  in  the  table,  it 
is  clear  that  during  the  next  decade  management  and  policy  makers 
are  going  to  face  some  serious  questions  if  these  projections 
are  to  materialize  by  applications  area. 


Table  3 

PROJECTION  OF  CRITERIA  USES  IN  EVALUATION  OF  MIS 
(BUbold,  1969) 


SATE 

SYSTEM  GENERATION 

MANAGEMENT  APPLICATION 

EVALUATION  CRITERIA  | 

1964 

Second 

Administration  and 
Accounting 

Clerical  Displacement;  cost 
reduction;  control 

1968 

Third 

Supervisory  Infotmatloo 

Reduction  In  Inventories;  re* 
duct! on  in  cash  balance; 
personnel  stability;  c  us  team  r 
relations;  vendor /buyer 
relations;  cost  control 

1975 

Third  ♦  and 

Fourth 

Middle  Management  and 
Tactical  Fleming 

Optima  car  hating  budget; 
return  on  short- tern  portfolios; 
iaproved  negotiating  position; 
{■proved  vendor  performance; 
optimum  nee  ot  plant;  Improved 
shipping  schedules;  mors 
realise  in  forecasting 

1  g 

Firth 

Top  oisiganint  and 
Strategic  F lancing 

Product  plaasUag;  capital 
requirements;  labor  and 
materials  planning;  resource 
p leaning;  transportation  and 
Inventory  planning. 
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5.  Conclusions 

The  development  of  management  information  systems  has  come  a 
long  way  in  the  last  ten  years,  heralded  much  by  advances  in 
technology.  The  computer,  however,  has  been  an  expensive  "fad" 
in  companies  where  management  has  not  taken  the  time  to  become 
involved  in  the  process  of  MIS  design.  The  milestone  called 
"evaluation"  is  the  most  conspicuous  and  weakest  stage  in  this 
process  today.  The  techniques  available  and  those  employed 
for  evaluation  need  much  improvement.  Perhaps  as  responses 
to  this  need  for  the  short  run  are  two  industry  trends.  The 
first  trend  in  evidence  is  the  emergence  within  large  corpora¬ 
tions  of  the  computer/information  systems  department  as  a 
"profit  center"  —  in  some  cases  incorporated  as  a  subsidiary, 
selling  services  to  outside  customers,  as  well.  The  second 
development  is  the  appearance  of  software  companies  with  the  . 
facilities  to  sell  ah  MIS  package  (not  limited  to  data 'management  . 
software)  to  a  customer  organisation.  The  vendor  in  this  case 
contracts  a  fixed-price-plus-maintenance  agreement  for  design, 
development,  implementation,  and  in  some  cases  continuing 
operation  (i.e,,  on  the  vendor's  equipment  or  under  an  installa¬ 
tion  management  arrangement).  I  think  both  trends  are  in  the 
right  direction,  but  represent  only  an  interim  solution  to  the 
evaluation  issue.  The  time  when  a  large  computer  installation 
is  "justified"  on  the  basis  on  intangibles  alone  is  passed.  There 
will  be  increasing  pressure  to  evaluate  MIS  proposals  on  their 
economic  potential  --  though  not:  on  administrative  cost  reduction, 
as  in  the  past.  However,  the  economics  of  upper  management 
decision  processes,  are  not  straight-forward  or  well  understood, 
and  many  questions  need  to  be  answered.  For  example,  what  economic 
and  behavioral  issues  should  be  detailed  for  an  MIS  In  support  of 
strategic  planning?  It  is  clear  to  me  ttiat  many  of  the  answers  must 
come  from  the  executives  involved. 
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As  a  professional  I  am  concerned  about  this  major  gap  in  the 
state-of-the-art  of  MIS,  particularly  when  I  see  and  feel  the 
push  of  technology  which  says:  Move!  1  suggest  we  all  have 
much  in  common  with  the  Red  Queen's  observation  to  Alice: 

"Now,  here,  you  see,  it  takes  all  the  running  you  can 
do,  to  keep  in  the  same  place.  If  you  want  to  get 
somewhere  else,  you  must  run  twice  as  fast  as  that" 

-(Through  the  Looking  Glass). 

I  urge  that  we  try  to  run  twice  as  fast  as  we  have,- not  in  develop¬ 
ing  the  brute  force  technology,  but  in  deciding  what  we  want  the 
technology  to  do. 

6.  References 


R.  A.  Arbuckle,  "Computer  Analysis  and  Thruput  Evaluation," 
Computers  and  Automation  (January  1966),  p.  13, 

J.  N.  Bairstow,  "A  Review  of  Systems  Evaluation  Packages," 

Computer  Decisions  (June  1970),  p.  20. 

J.  Becker  and  R.  M.  Hayes,  Information  Storage  and  Retrieval: 
Tools.  Elements,  Theories  (John  Wiley  and  Sonr,  1963). 

B.  F.  Boyu  and  H.  S.  Krasnov,  "Economic  Evaluation  of  Management 

Information  Systems,"  IBM  Systems  Journal  (March  1963), 
p.  2-23. 

t 

A.  C.  Bromley, "Choosing  a  Set  of  Computers,"  Datamation 
(August  1965),  p.  37-40. 

W.  Buchholz,  "A  Selected.  Bibliography  on  Computer  System  Perfor¬ 
mance  Evaluation,"  Computer  Group  News  I.E.E.E.  (March  1969), 

p.  21-22. . 

C.  J.  Byrnes  and  D.  B.  Steig,  "File  Management  Systems:  A  Current 

Summary,"  Datamation  (November  1969),  p.  138-142. 

P.  Calingacrt,  "System  Performance  Evaluation:  Survey  and  Ap¬ 
praisal,"  Communications  of  the  ACM  (January  1967),  p.  12-18. 

C.  West  Churchman,  The  Systems  Approach  (Delacorte  Press,  1968). 


I 


20- 


CODASYL  Systems  Committee,  "A  Survey  of  Generalized  Data  Base 
Management  Systems"  (May  1969),  Association  for  Computing 
Machinery,  New  York  City. 

/ 

CODASYL  Data  Base  Task  Group,  "October  1969  Report",  Association 
for  Computing  Machinery,  New  York  City. 

John  Diebold,  "Bad  Decisions  on  Computer  Use,"  Harvard  Business 
Review  (January-February  1969),  p.  14-16  ff. 

J.  Fry  and  J.  Gosden,  "Survey  of  Management  Information  Systems 

and  Their  Languages,"  p.  41-56  in  F.  Gruenberger  (ed.), 

Critical  Factors  in  Data  Management  (Prentice-Hall,.  1969). 

R.  V.  L.  Hartley,  "Transmission  of  Information,"  Bell  System 
Technical  Journal.  Vol.  7  (1928),  p,  535-563. 

P.  B.  Henderson,  Jr.,  "A  Theory  of  Data  Systems  for  Economic 
Decisions,"  unpublished  Ph.D.  Thesis,  Economics  Department, 
M.I.T.,  Cambridge,  Mass.  (June  1960). 

D.  J.  Herman  and  F.  C.  Ihrer,  "The  Use  of  a  Computer  to  Evaluate 
Computers,"  Proceedings  AFIPS  1964  Spring  Joint  Computer 
Conference.  Vol.  25,  (April  1964),  p.  383-395. 

John  R.  Hillegass,  "Standardized  Benchmark  Problems  Measure 
Computer  Performance,"  Computers  and  Automation  (January 
1966),  p.  16-19.  i 

IBM  Corp.  (1963).  Publications  on  "Study  Organization  Plan"  (SOP): 
The  Approach.  F  20-8135;  Documentation  Techniques.  C20-8075-0; 
The  Method  Phase  I.  F20-8036-0;  The  Method  Phase  II.  F20-8137; 
The  Method  Phase  III.  F20-8138-0. 

IBM:  "Systems  360  Model  67  Time-Sharing  System  Technical  Summary" 
(August  18,  1965). 

K.  E.  Knight,  "A  Study  of  Technological  Innovation  —  The  Evolution 

of  Digital  Computers,"  unpublished  Ph.D.  Thesis,  Graduate 
School  of  Industrial  Administration,  Carnegie-Mellon  University 
(November  1963). 

C.  H.  Kriebcl,  "Operations  Research  in  the  Design  of  Management 
Information  Systems,"  Chapter  22,  375-390  in  J.  F.  Pierce 
(ed.),  Operations  Research  and  the  Design  of  Management 
Information  Systems  (Technical  Association  of  the  Pulp  and 
Paper  Industry,  1967). 

C.  H.  Kricbcl,  "The  Strategic  Dimension  of  Computer  Systems 
Planning,"  Long  Range  Planning  (September  1968),  p.  7-12. 


-22- 


C.  H.  Kriebel,  J.  T.  Hcatnes ,  and  R.  L.  Van  Horn  (ed.),  Manage¬ 
ment  Information  Systems:  Progress  and  Perspectives 
(Carnegie-Mellon  University,  1970). 

R.-  M.  Laska,  "Keeping  Your  Computer  Out  of  the  Till",  Computer 
Decisions  (May  1970),  p.  19-23. 

Jacob  Marschak,  "Flements  for  a  Theory  of  Teams,"  Management 
Science  (January  1955),  p.  127-137. 

J.  Marschak,  "Economics  of  Inquiring,  Communicating,  Deciding," 
American  Economic  Review  (May  1968),  p.  1-18, 

McKinsey  and  Company,  Inc.,  Unlocking  the  Computer's  Profit 

Potential  (New  York  City,  1968);  reprinted  in  Computers  and 
Automation  (April  1969),  p.  24-33. 

"Computer  Evaluation  and  Selection:  A  Bibliography,"  prepared  for 
the  Association  for  Computing  Machinery  by  Metametrics  Corpora¬ 
tion  (1969);  ACM  Professional  Development  Seminar.  (61  entries). 

H.  Raiffa,  Decision  Analysis  (Addison-Wesley ,  1968). 

Roy  Radner,  "Team  Decision  Problems,"  Annals  of  Mathematical 
Statistics  (September  1962),  p.  857-881. 

N.  F.  Schneidewind,  "Analytical  Model  for  the  Design  and  Selection 
of  Electronic  Digital  Computers,"  Unpublished  D.B.A.  Thesis, 
Graduate  School  of  Business,  University  of  Southern  California 
(January  1966). 

C,  E.  Shannon  and  W.  Weaver,  The  Mathematical  Theory  of  Communica¬ 
tion  (University  of  Illinois  Press,  1948). 

William  F.  Sharpe,  The  Economics  of  Computers  (Columbia  University 
Press,  1969). 

H,  A.  Simon,  The  Sciences  of  the  Artificial  (MIT  Press,  1969). 

M.  B.  Solomon,  Jr.,  "Economies  of  Scale  and  the  IBM  System/360," 
Communications  of  the  ACM  (June  1966),  p,  435-440, 


